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Abstract — We consider stability and network capacity in dis- 
crete time queueing systems. Relationships between four common 
notions of stability are described. Specifically, we consider rate 
stability, mean rate stability, steady state stability, and strong 
stability. We then consider networks of queues with random 
events and control actions that can be implemented over time 
to affect arrivals and service at the queues. The control actions 
also generate a vector of additional network attributes. We 
characterize the network capacity region, being the closure of the 
set of all rate vectors that can be supported subject to network 
stability and to additional time average attribute constraints. 
We show that (under mild technical assumptions) the capacity 
region is the same under all four stability definitions. Our 
capacity achievability proof uses the drift-plus-penalty method 
of Lyapunov optimization, and provides full details for the case 
when network states obey a decaying memory property, which 
holds for finite state ergodic systems and more general systems. 

Index Terms — Queueing analysis, opportunistic scheduling, 
flow control, wireless networks 



I. Introduction 

This paper considers stability and network capacity in dis- 
crete time queueing systems. These issues arise in the analysis 
and control of modern data networks, including wireless and 
ad-hoc mobile networks, and are also important in many other 
application areas. We have found that many researchers have 
questions about the relationships between the different types 
of stability that can be used for network analysis. This paper 
is written to address those questions by providing details on 
stability that are mentioned in other papers but are not proven 
due to lack of space. 

We consider the four most common types of stability from 
the literature: rate stability, mean rate stability, steady state 
stability, and strong stability. We first show that, under mild 
technical assumptions, strong stability implies the other three, 
and hence can be viewed as the strongest definition among 
the four. Conversely, we show that mean rate stability is the 
weakest definition, in that (under mild technical assumptions) 
it is implied by the other three. We also briefly describe 
additional stability definitions, such as existence of a steady 
state workload distribution as in [1] [2] [3] (often analyzed with 
Markov chain theory and Lyapunov drift theory [4] [5] [6] [7] 
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and/or fluid models [8] [9]), and discuss their relationships to 
the main four. 

We then consider control for a general stochastic multi- 
queue network. The network operates in discrete time with 
timeslots t e {0, 1,2,.. .}. Control actions are made on each 
slot in reaction to random network events, such as random 
traffic arrivals or channel conditions. The control actions and 
network events affect arrivals and service in the queues, and 
also generate a vector of network attributes, being additional 
penalties or rewards associated with the network (such as 
power expenditures, packet drops, etc.). We assume the system 
satisfies the mild technical assumptions needed for the above 
stability results. We further assume that network events have a 
decaying memory property, a property typically exhibited by 
finite state ergodic systems as well as more general systems. 
As in [7], we define the network capacity region A as the 
closure of the set of all traffic rate vectors that can be supported 
subject to network stability and to an additional set of time 
average attribute constraints. We show that if traffic rates are 
outside of the set A, then under any algorithm there must be at 
least one queue that is not mean rate stable. Because mean rate 
stability is the weakest definition, it follows that the network 
cannot be stable under any of the four definitions if traffic rates 
are outside of A. Conversely, we show that if the traffic rate 
vector is an interior point of the set A, then it is possible to 
design an algorithm that makes all queues strongly stable (and 
hence it is also possible to achieve stability for the other three 
stability definitions). Because the capacity region is defined as 
a closure, it follows that it is invariant under any of these four 
stability definitions. 

As an example, consider a simple discrete time GI/GI/1 
queue with fixed size packets and arrivals a(t) that are i.i.d. 
over slots with E{a(£)} = A packets/slot, and independent 
time varying service rates b(t) that are i.i.d. over slots with 
E {b(t)} = 1/2 packets/slot. The "mild technical assumptions" 
that we impose here are that the second moments of the a(t) 
and b(t) processes are finite. In this setting, the capacity region 
is the set of all arrival rates A such that < A < 1/2. It turns 
out that the queue is rate stable (and mean rate stable) if and 
only if A < 1/2. However, for steady state stability and strong 
stability we typically require A < 1/2 (with an exception in 
certain deterministic special cases). Thus, the set of all rates 
for which the queue is stable differs only by one point (the 
point A = 1/2) under the four different definitions of stability, 
and the closure of this set is identical for all four. There are 
indeed alternative (problematic) definitions of "stability" that 
would give rise to a different capacity region, although these 
are typically not used for networks and are not considered here 
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(see an example in [10] of a problematic definition that says a 
queue is "empty-stable" if it empties infinitely often, and see 
Section HV-Bl for another problematic example). 

The above 1-queue example considers a(t) and bit) pro- 
cesses that are i.i.d. over slots. However, our stability and 
capacity region analysis is more general and is presented 
in terms of processes that are possibly non-i.i.d. over slots, 
assuming only that they have well defined time averages 
with a mild "decaying memory" property. We show that the 
capacity region is achievable using a strong drift-plus-penalty 
method that we derived in previous papers [10][7][1 1] [12]. 
This method treats joint stability and performance optimiza- 
tion, and extends the pioneering work on network stability in 
[13] [14]. These results are easier to derive in a context when 
network arrival and channel vectors are i.i.d. over slots. The 
prior work on network stability in [15] treats general Markov- 
modulated channels. Arrivals and channels with a "decaying 
memory property" are treated for stability in [16] [10]. Joint 
stability and utility optimization are considered for non-i.i.d. 
models in [17][18][19][20] for different types of networks. 
This paper provides the full details for the non-i.i.d. case of 
joint stability and utility optimization for general networks 
with general time average constraints. A more general "uni- 
versal scheduling" model is treated in [21], which uses sample 
path analysis and considers possibly non-ergodic systems with 
no probability model, although the fundamental concept of a 
"capacity region" does not make sense in such a context. 

II. Queues 

Let Q(t) represent the contents of a single discrete 
time queueing system defined over integer timeslots t G 
{0, 1,2,.. .}. Specifically, the initial state Q(0) is assumed to 
be a non-negative real valued random variable. Future states 
are driven by stochastic arrival and server processes a(t) and 
b(t) according to the following dynamic equation: 

Q(t + 1) = max[Q(<) - b(t), 0] + a(t) for t e {0, 1,2,.. .} 

(1) 

We call Q(t) the backlog on slot t, as it can represent an 
amount of work that needs to be done. The stochastic pro- 
cesses {a(t)}^ Q and {b(t)}^L are sequences of real valued 
random variables defined over slots t £ {0, 1,2,.. .}. 

The value of a(t) represents the amount of new work that 
arrives on slot t, and is assumed to be non-negative. The value 
of b(t) represents the amount of work the server of the queue 
can process on slot t. For most physical queueing systems, 
b(t) is assumed to be non-negative, although it is sometimes 
convenient to allow b(t) to take negative values. This is useful 
for the virtual queues defined in future sections, where b(t) 
can be interpreted as a (possibly negative) attribute^] Because 
we assume Q(Q) > and a(t) > for all slots t, it is clear 
from 0} that Q(t) > for all slots t. 

'Assuming that the b(t) value in (T} is possibly negative also allows 
treatment of modified queueing models that place new arrivals inside the 
max[-,0] operator. For example, a queue with dynamics Q(t + 1) = 
max[Q(t) - /3(t) + a(t),0] is the same as {TJ with a(t) = and 
b(t) = j3[t) — a(t) for all t. Leaving a(t) outside the max[-,0] is crucial 
for treatment of multi-hop networks, where a(t) can be a sum of exogenous 
and endogenous arrivals. 



The units of Q{t), a{t), and b{t) depend on the context 
of the system. For example, in a communication system with 
fixed size data units, these quantities might be integers with 
units of packets. Alternatively, they might be real numbers 
with units of bits, kilobits, or some other unit of unfinished 
work relevant to the system. 

We can equivalently re-write the dynamics CQl without the 
non-linear max[-,0] operator as follows: 

Q(t+l) = Q(t)-b(t) + a(t) fori G {0,1,2,...} (2) 

where b(t) is the actual work processed on slot t (which 
may be less than the offered amount b(t) if there is little 
or no backlog in the system at slot t). Specifically, b(t) is 
mathematically defined: 

b(t)Amm[b(t),Q(t)} 

Note by definition that b(t) < b(t) for all t. The dynamic 
equation (fJJ yields a simple but important property for all 
sample paths, described in the following lemma. 

Lemma 1: (Sample Path Property) For any discrete time 
queueing system described by ([TJ, and for any two slots t\ 
and t2 such that < t\ < t^, we have: 

t 2 -i t 2 -i 



Q(b)-Q(h) = £ o(r) - £ 6(r) (3) 

T — t-i T — ti 

t 2 ~l t 2 ~l 



r=ti 



Therefore, for any t > we have: 

t-i 

t 



Q(t) Q(0) 



t-i 



t 



t 



Q(t) Q(0) 



= tI>m-7E^) (5) 

T=0 r=0 
t-1 i-1 



(6) 



t t - t ^ v ' t 

T=0 T=0 

Proof: By (ffj we have for any slot r > 0: 

Q(t + 1) - Q(t) = o(r) - 6(t) 

Summing the above over r € {t\, . . . , ti — 1} and using 
telescoping sums yields: 

ta-l i a -l 

0(*a) - Q(h) = £ a(r) - £ b(r) 

T — t± T — ti 

This proves (0. Inequality (0]) follows because b(r) < b(r) 
for all r. Inequalities © and © follow by substituting t\ = 0, 
t-i = t, and dividing by t. □ 
The equality (T5J) is illuminating. It shows that Q(t)/t —> 
as t — > oo if and only if the time average of the process 
a(t) — bit) is zero (where the time average of a(t) — bit) is 
the limit of the right hand side of ©). This happens when the 
time average rate of arrivals a(t) is equal to the time average 
rate of actual departures This motivates the definitions 
of rate stability and mean rate stability, defined in the next 
section. 
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III. Rate Stability 

Let Q(t) be the backlog process in a discrete time queue. 
We assume only that Q(t) is non-negative and evolves over 
slots t G {0, 1,2,...} according to some probability law0 

Definition 1: A discrete time queue Q(t) is rate stable if: 



lim 



Q(t) 



with probability 1 



t— >oo t 

Definition 2: A discrete time queue Q{t) is mean rate 
stable if: 

t— >oo t 

Neither rate stability nor mean rate stability implies the 
other (see counter-examples given in Section [V). However, 
rate stability implies mean rate stability under the following 
mild technical assumptions. 

Theorem 1: (Rate Stability & Bounding Assumptions Im- 
plies Mean Rate Stability) Consider a queue Q(t) with dy- 
namics (fl}, with b(t) real valued (possibly negative) and a(t) 
non-negative. Suppose that Q(t) is rate stable. 

a) Suppose there are finite constants e > and C > 
such that E{(a(t) +b~(t)) 1+e } < C for all t, where b~(t) 
is defined: 



6-(t)4-min[6(t),0] 



(7) 



Then Q(t) is mean rate stable. 

b) Suppose there is a non-negative random variable Y with 
E {Y} < oo and such that for all t e {0, 1,2,.. .} we have: 

E {a(t) + b-(t)\a(t) + b~(t) > y) Pr[a{t) + b~(t) > y] 

< E{Y\Y > y}Pr[Y > y] Vy G R (8) 

Then Q(t) is mean rate stable. 

Proof: See Appendix A. □ 

We note that the condition (0 holds whenever the random 
variable Y is stochastically greater than or equal to the 
random variable a(t) + b~ (t) for all t [23]. This condition also 
trivially holds whenever a(t) + b~(t) is stationary, having the 
same probability distribution for all slots t but not necessarily 
being i.i.d. over slots, and satisfies E {a(0) + b~ (0)} < oo. 
This is because, in this stationary case, we can use Y = 
a(0) + b~(0). The condition in part (a) does not require 
stationarity, but requires a uniform bound on the "(1 + e)" 
moment for some e > 0. This certainly holds whenever the 
second moments of a(t) + b~(t) are bounded by some finite 
constant C for all t (so that e = 1), as assumed in our network 
analysis of Section [VTl 

The next theorem gives intuition on rate stability and mean 
rate stability for queues with well defined time average arrival 
and server rates. 

Theorem 2: (Rate Stability Theorem) Suppose Q(t) evolves 
according to (|T), with a(t) > for all t, and with b(t) real 
valued (and possibly negative) for all t. Suppose that the 
time averages of the processes a(t) and b(t) converge with 

2 A11 of our stability definitions can be extended to treat discrete time 
stochastic processes Q(t) that can possibly be negative by substituting |Q(t) 
into the definitions, which is sometimes useful in contexts (not treated here) 
where virtual queues can be possibly negative, as in [21] [22]. 



probability 1 to finite constants a av and b av , so that: 
1 t_1 

lim - a(r) = a av with probability 1 (9) 

t— ¥00 t — J 



T=0 
t-1 



lim - b(r) — b av with probability 1 (10) 

t->oo t ^ — ' 



T=0 

Then: 

(a) Q(t) is rate stable if and only if a av < b av . 

(b) If a av > b av , then: 

Q(t) 

lim = aav — b m with probability 1 

t— >oo t 

(c) Suppose there are finite constants e > and C > 
such that E{(a(f) +b~(t)) 1+t } < C for all t, where b~(t) 
is defined in (0. Then Q(t) is mean rate stable if and only if 

(lav — bav- 

(d) Suppose there is a non-negative random variable Y with 
E {Y} < oo and such that condition <[8J holds for all t G 
{0,1,2,...}. Then Q(t) is mean rate stable if and only if 

(Lav — bav- 

Proof: (Theorem |2]i Suppose that Q(t) is rate stable, so 
that Q(t)/t — > with probability 1. Because © holds for 
all slots t > 0, we can take limits in (|6]l as t — >• oo and use 
<f9b-(fT0l> to conclude that > a av ~ b av - Thus, a av < b av is 
necessary for rate stability. The proof for sufficiency in part 
(a) and the proof of part (b) are not obvious and are developed 
in Exercises [2] and [3] of Section IVIIII 

To prove parts (c) and (d), suppose that a av < b av . We thus 
know by part (a) that Q(t) is rate stable. The conditions in 
parts (c) and (d) of this theorem correspond to the conditions 
given in Theorem Q] and hence Q(t) is mean rate stable. 

Now suppose that a av > b av - It follows by part (b) that: 



km = a a 

t—too t 



— b av with prob. 1 



Define 8={a av — b av )/2. Note that: 



Therefore: 



lim Pr[Q(t)/t > 5} = 1 

t— >oo 



E< fQW|^) >S }Pr[Q(t)/t>5] 



> 5Pr[Q(t)/t> 5} 



Taking a limit yields: 



lim sup E < > > d 

t^oo I, t J 

and hence Q(t) is not mean rate stable. □ 
Prior sample path investigations of constant service rate 
queues are provided in [24] [25] [26] [2], where it is shown 
that rate stability holds whenever the arrival rate is strictly 
less than the service rate. Our proof of Theorem |2ja) uses a 
different chain of reasoning (developed in Exercises [2] and [3] 
of Section IVIIIl i, applies to queues with more general time 
varying (and possibly negative) service rates, and also shows 
the case a av — b av ensures rate stability, which establishes the 
simple necessary and sufficient condition a av < b av - 
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The assumption that ait) and b(t) have well defined time 
averages a av and b av is crucial for the result of Theorem [2] 
One might intuitively suspect that if a(t) has a well defined 
time average a av , but bit) has limsup and liminf time 
averages b™J and b s a ^ p such that a av < b™J < b s ^ p , then Q(t) 
is also rate stable. This is not always true. Thus, the existence 
of well defined time averages provides enough structure to 
ensure queue sample paths are well behaved. The following 
theorem presents a more general necessary condition for rate 
stability that does not require the arrival and server processes 
to have well defined time averages. 

Theorem 3: (Necessary Condition for Rate Stability) Sup- 
pose Q(t) evolves according to ((T), with any general processes 
a(t) and b(t) such that a(t) > for all t. Then: 

(a) If Qit) is rate stable, then: 

1 

limsup - ^~^[q(r) — 6(r)] < with probability 1 (11) 

t— ► oo t 

r=0 

(b) If Qit) is mean rate stable and if E{Q(0)} < oo, then: 

1 * _1 

limsup - E{a(r) - 6(r)} < (12) 

Proof: The proof of (a) follows immediately by taking 
a limsup of both sides of © and noting that Q(t)/t —> 
because Q(t) is rate stable. The proof of (b) follows by first 
taking an expectation of (O and then taking limits. □ 

IV. Stronger Forms of Stability 

Rate stability and mean rate stability only describe the 
long term average rate of arrivals and departures from the 
queue, and do not say anything about the fraction of time 
the queue backlog exceeds a certain value, or about the time 
average expected backlog. The stronger stability definitions 
given below are thus useful. 

Definition 3: A discrete time queue Q(t) is steady state 
stable if: 

lim g(M) = 
where for each M > 0, g(M) is defined: 

1 

S(M)4 lim sup -^Pr[Q(r) > M] (13) 



T = 



Definition 4: A discrete time queue Q(t) is strongly stable 



if: 



1 t_1 

limsup - ^E{Q(t)} < oo 

t— ► oo t 



(14) 



r=0 



For discrete time ergodic Markov chains with countably 
infinite state space and with the property that, for each real 
value M, the event {Qit) < M} corresponds to only a finite 
number of states, steady state stability implies the existence of 
a steady state distribution, and strong stability implies finite 
average backlog and (by Little's theorem [3]) finite average 
delay. Under mild boundedness assumptions, strong stability 
implies all of the other forms of stability, as specified in 
Theorem [4] below. 

Theorem 4: (Strong Stability Theorem) Suppose Qit) 
evolves according to (Q~|i for some general stochastic processes 



{a(t)}^ and {b(t)}^ , where a(t) > for all t, and b(t) is 
real valued for all t. Suppose Q(t) is strongly stable. Then: 

(a) Qit) is steady state stable. 

(b) If there is a finite constant C such that either a(t) + 
b~(t) < C with probability 1 for all t (where is defined 
in ©), or b{t) - a(t) < C with probability 1 for all t, then 
Q{t) is rate stable, so that Q(t)/t —> with probability 1. 

(c) If there is a finite constant C such that either 
K{a{t) + b-(t)} < C for all t, or E{6(t) - a(t)} < C for 
all t, then Qit) is mean rate stable. 

Proof: Part (a) is given in Exercise [4] Part (c) is given in 
Appendix B, and part (b) is given in Appendix C. □ 

The above theorem shows that, under mild technical as- 
sumptions, strong stability implies all three other forms of 
stability. Theorem Q] and Theorem |4jc) show that (under mild 
technical assumptions) rate stability and strong stability both 
imply mean rate stability. For completeness, the following 
theorem provides conditions under which steady state stability 
implies mean rate stability. Collectively, these results can 
be viewed as showing that strong stability is the strongest 
definition of the four, and mean rate stability is the weakest 
definition of the four. 

Theorem 5: Assume Qit) evolves according to (T} with 
ait) > and bit) real values for all t. Suppose that Qit) 
is steady state stable, and that there is a finite constant C such 
that a(t) + b~(t) < C with probability 1 for all t. Then Qit) 
is mean rate stable. 

Proof: See Appendix D. □ 

A. Sample Path Versions of Stability 

One might use a sample-path version of strong stability, 
saying that a queue is sample-path strongly stable if: 

1 * _1 

limsup - Qi T ) < oo with prob. 1 

t— ¥OQ t 

T = 

A sample-path version of steady-state stable would re-define 
the function g(M) in ( fT3] l by a function ft,(M) as follows: 

1 

/i(M)4 limsup - ^ 1{Q(t) > M} 

where 1{Q(t) > Al} is an indicator function that is 1 if 
Q(t) > M, and zero otherwise. We might say that the queue 
is sample-path steady-state stable if lim^/^oo /i(M) = with 
probability 1. These two additional stability definitions are 
again implied by strong stability if one assumes the system 
has well defined limits (which is typically the case in systems 
defined on Markov chains), as shown in Appendix E. 

B. A Problematic Stability Definition 

Finally, one might define another form of stability by 
requiring: 

limsupE{Q(X)} < oo (15) 

t— >OQ 

It is clear that if E {Qit)} < oo for all t and if the above holds, 
then the time average of E{Q(i)} is also finite and so Qit) 
is strongly stable. Hence, the condition (TT3T > can be viewed 
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as being even "stronger" than strong stability. Of course, in 
most systems defined over Markov chains, strong stability is 
equivalent to ( fl~5T >. 

However, we do not consider (fl3T l in our set of stability 
definitions for two reasons: 

1) The strong stability definition that uses a time average 
is easier to work with, especially in Lyapunov drift 
arguments [7]. 

2) The condition ( fT5l ) leads to a problematic counterexample 
if it were used as a definition of stability, as described 
below. 

To see why the definition ( TT~5T > may be problematic, con- 
sider a simple discrete time "Bernoulli/Bernoulli/1" (B/B/l) 
queue, where arrivals a(t) are i.i.d. over slots with Pr[a(t) = 
1] = A and Pr[a(t) = 0] = 1 — A, and the server process b(t) 
is independent and i.i.d. over slots with Pr[b(i) = 1] = /i, 
Pr[b(t) = 0] = 1 — fi. When A < fi, it is easy to write 
the ergodic birth-death Markov chain for this system, and one 
can easily derive that the Markov chain has a well defined 
steady state, steady state probabilities decay geometrically, and 
average queue backlog and delay satisfy: 

/! — A /i — A 

Thus, if A < /i, a good definition of stability would say that 
this system is stable. 

Now suppose we take one particular sample path realization 
of the B/B/l queue, one for which time averages converge 
to the steady state values (which happens with probability 
1). However, treat this sample path as given, so that all 
events are now deterministic. Thus, a(t) and b(t) are now 
deterministic functions. Because we have not changed the 
actual sample path, a good definition should also say this 
deterministic variant is stable. In this deterministic case, we 
have E{Q(t)} = Q(t) for all t, and so the expectation 
can grow arbitrarily large (since a B/B/l queue can grow 
arbitrarily large). Hence, for this deterministic example: 

limsupE{Q(t)} = oo 

t— >oo 

Thus, if we used ( fT3T > as a definition of stability, the random 
B/B/l queue would be stable, but the deterministic version 
would not! Another way of saying this is that the original 
B/B/l queue is stable, but if we condition on knowing all 
future events, it becomes unstable! 

Because our definitions of rate stability, mean rate stability, 
sample path stability, and strong stability incorporate time 
averages, these four forms of stability all say that both the 
random B/B/l queue and its deterministic counterpart are 
stable. Hence, the problem does not arise for this example in 
any of these four definitions. 

Another problematic stability definition says that a queue is 
"empty-stable" if it empties infinitely often. A discussion of 
why this is problematic is provided in [10]. 

V. COUNTER-EXAMPLES 

Here we provide counter-examples that show what can 
happen if the boundedness assumptions of Theorems [T] and 
|4]are violated. 



A. Rate Stability Does Not Imply Mean Rate Stability 

Let T be an integer random variable with a geometric 
distribution, such that Pr[T > t] = 1/2* for t G {0, 1,2,.. .}. 
Define Q(t) over t <E {0, 1,2,.. .} as follows: 



Q(t) = 



2 (2t) if t <T 







otherwise 



It follows that Q(t)/t with probability 1 (as eventually t 
becomes larger than the random variable T). However: 

E{Q(t)} = 2 [2t) Pr[t <T] = 2 (2 * ) 2~ t = 2* 

Therefore: 

lim E{Q(t)} It = lim 2* It = oo 

t— >oo t— ^oo 

and hence Q(t) is not mean rate stable. 

B. Mean Rate Stability Does Not Imply Rate Stability 

Suppose that Q(0) = 0, and that Q(t) has independent 
values every slot t for t € {1, 2,3,.. .}, so that: 



Q(t) = 



t with probability 1/t 



with probability 1 — 1/t 
Thus, for any time t > we have: 

E{Q(t)} =t/t=l 

It follows that E{Q(t)} /t -> 0, and so Q(t) is mean rate 
stable. 

However, clearly Q(t)/t = 1 for any time t such that 
Q(t) > 0. Because the probabilities of the independent events 
at which Q(t) > decay very slowly, it can be shown that 
there are an infinite number of times t n for which Q(t n ) > 0. 
Indeed, we have for any time t > 0: 

oo 

Pr[Q{r) = for all r > t] = JJ(1 - 1/t) = 

The infinite product can be shown to be zero for any t > 
by taking a log(-) and showing that the resulting infinite sum 
is equal to — oo. Therefore: 

Q(t) 

lim sup = 1 with probability 1 

t— s-oo t 

and so the system is not rate stable. 

C. Strong Stability Neither Implies Mean Rate Stability Nor 
Rate Stability 

Suppose that Q(t) — t for t £ {1, 2, 4, 8, ... , 2", . . .} (i.e., 
for all timeslots t that are powers of 2). Suppose that Q(t) = 
at all slots t that are not powers of 2. Because this process 
is deterministic, we have E{Q(t)} = Q{t), and for all n E 
{0, 1,2,.. .} we have: 



2" 

T=0 



Q(r) 



2" 2" +1 - 1 
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The right hand side of the above expression converges to 2 
as n — > oo. It can be shown that j Y^t=o Q( t ) ^ s ^ e lar g est 
when sampled at the times in the above expression, and hence: 




Because E{C](t)} = Q(r) for all r, it follows that: 

1 4-1 

limsup-^E{Q(r)} = 2 

Therefore, Q(t) is strongly stable. However, Q(2")/2™ = 1 
for all n G {1,2,...}, and so Q(t) is not rate stable or 
mean rate stable (rate stability and mean rate stability are 
equivalent when Q(t) is deterministic). Note in this case 
that increases or decreases in Q(t) can be arbitrarily large, 
and hence this example does not satisfy the boundedness 
assumptions required in Theorem 0] 

VI. Network Scheduling 

Consider now the following multi-queue network model 
defined over discrete time t € {0,1,2,...}. There are K 
queues Q(t) = (Qi{t), . . . ,Qx(t)), with dynamics for all 
ke{l,...,K}: 

Qk(t + 1) = Q k {t) - k(t) + y k {t) + a k {t) (16) 

where ak(t) represents new exogenous arrivals to queue fc 
on slot t, bk(t) represents the actual amount served on slot t, 
and y k (t) represents additional arrivals. The additional arrivals 
yk(t) may be due to flow control operations (in which case 
we might have dk(t) = so that all new arrivals are first 
passed through the flow control mechanism). They might also 
be due to endogenous arrivals from other queues, which allows 
treatment of multi-hop networks. 

We assume the queue is always non-negative, as are b k (t), 
Vk(t), a k (t), and that the bk(t) and y k (t) values respect the 
amount of data that is actually in each queue (not serving 
more or delivering more than the amount transferred over the 
channel). It is also useful to assume transmission decisions 
can be made independently of queue backlog, and so we also 
define bk(t) and y k (t) for queue dynamics: 

Q k {t + 1) < max[Q k {t) ~ b k {t),0] + y k (t) + a k {t) (17) 

The inequality is due to the fact that the amount of actual 
new exogenous arrivals y k (t), being a sum of service values 
in other queues that transmit to queue fc, may not be as large 
as yk(t) if these other queues have little or no backlog to 
transmit. These values satisfy: 

< y k (t) < y k (t) Vfce{l,...,X},W 
o< h(t) < b k (t) v*e{i,...,Jr},vt 

This model is similar to that given in [7] [16] [10], and the 
capacity region that we develop is also similar to prior work 
there. 

The network has a time varying network state w(t), possibly 
being a vector of channel conditions and/or additional random 
arrivals for slot t. A network control action a(t) is chosen 



in reaction to the observed network state w(t) on slot t (and 
possibly also in reaction to other network information, such as 
queue backlogs), and takes values in some abstract set A^U) 
that possibly depends on ui(t). The ui(t) and a(t) values for 
slot t affect arrivals and service by: 

y k (t) = y k (a(t),u(t)) Vfee {h...,K} 
b k (t) = bk(a(t),iu(t)) Wke{l,...,K} 

where y k (a(t),uj(t)) and bk(a(t),uj(t)) are general functions 
of ait) and u(t) (possibly non-convex and discontinuous). 
The ui(t) and a(t) values also affect an attribute vector 
x{t) = (xi(t), . . . , imW) for slot t, which can represent 
additional penalties or rewards associated with the network 
states and control actions (such as power expenditures, packet 
admissions, packet drops, etc.). The components x m (t) can 
possibly be negative, and are general functions of a(t) and 
w(t): 

x m (t) = x m (a(t),uj(t)) 

A. Network Assumptions 

We assume that exogenous arrivals a k {t) satisfy: 
t-i 

lim - VE{a fc (r)} = A fc Vfc G {1, . . . , if} (18) 

t— >oo t A — ' 

T = 

where we call Afe the arrival rate for queue fc. We assume the 
network state ui(t) is stationary with a well defined stationary 
distribution ir(ui). In the case when there are only a finite or 
countably infinite number of network states, given by a set 
il, then it(uj) represents a probability mass function and by 
stationarity we have: 

tt(cj) = Pr[u)(t) = uj] Vw e Q,Vt e {0,1,2,...} 

In the case when 17 is possibly countably infinite, then we 
assume tu(t) is a random vector and tt(uj) represents a prob- 
ability density function. The simplest model is when u(t) is 
i.i.d. over slots, although the stationary assumption does not 
require independence over slots. 

We further assume that the control decision a(t) € A u m 
can always be chosen to respect the backlog constraints, and 
that any algorithm that does not respect the backlog constraints 
can be modified to respect the backlog constraints without 
hindering performance. This can be done simply by never 
attempting to transmit more data than we have, so that for 
all fc g {1, . . . , K} we have: 

b k (a(t),oj(t)) = b k (t) = b k (t) (19) 
y k {a{t),u{t))=y k (t)=y k (t) (20) 

We define a policy a(t) that satisfies (fl9ll-(l20li to be a policy 
that respects queue backlog. It is clear that the queue dynamics 
(fi7] i under such a policy become: 

Qfc(t+1) = Qk(t)-h(t)+y k (t)+a k (t)Vk € {l,...,K},Vt 

(21) 
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B. The Optimization Problem 

Let f(x) and gi(x),g2(x), . . . ,gL(x) be real-valued, con- 
tinuous, and convex functions of x £ M M for some non- 
negative integer L (if L = then there are no <#(a;) 
functions)]! Suppose we want to design a control algorithm 
that chooses a(t) £ A u it) over slots t that solves the following 
general stochastic network optimization problem: 

Minimize: limsup f(x(t)) (22) 

t— >oo 

Subject to: 1) limsup gi (x(t)) < V/ G {1, ... ,L} (23) 

t— >oo 

2) a(t)£A u{t) Vie {0,1,2,...} (24) 

3) All queues Qk{t) are mean rate stable (25) 

where x(t) is defined for t > by: 



^)4i>{^)} 



T = 

We say that the problem is feasible if there exists a control 
algorithm that satisfies the constraints d23l-(l25]l. Assuming the 
problem is feasible, we define f opt as the infimum cost in ( 1221 
over all possible feasible policies that respect queue backlog. 

We define an u-only policy as a policy that observes uj(t) 
and makes a decision a(t) £ A u (t) as a stationary and random 
function only of u>(t) (regardless of queue backlog, and hence 
not necessarily respecting the backlog constraints (fT9l)-(|20l)). 
By stationarity of ui(t), it follows that the expected values of 
bk(t), Vk(t) x m (t) are the same on each slot under a particular 
w-only policy a*(t) £ A u ^y. 



Vk 



E{S fc (a*(t),w(t))} 
E{y k (a*(t),Lu(t))} 
E{x m (a*(t),u(t))} 



where the expectation above is with respect to the stationary 
distribution tt(lu) and the possibly randomized actions a*(t). 
The next theorem characterizes all possible feasible algorithms 
(including algorithms that are not w-only) in terms of w-only 
algorithms. 

Theorem 6: Suppose the problem (f22l>-(|25T> is feasible with 
infimum cost f opt , assumed to be achievable arbitrarily closely 
by policies that respect the backlog constraints (fT~9T> - (l2Qb . 
Then for all e > there exists an w-only algorithm a*(t) 
that satisfies the following for all k £ {1,...,K} and I £ 
{1 '■}■ 

gi (E{x(a*(t),Lu(t))})<e (26) 
X k +E{y k (a*(t),u;(t))-b k (a*(t)Mt))} <e (27) 

f(E{x(a*(t)Mt))})<f° pt + ^ (28) 
Before proving Theorem|6] we present a preliminary lemma. 

Lemma 2: For any algorithm that chooses a(r) £ A^ir) 
over slots r £ {0, 1,2,...,}, and for any slot t > 0, there 

3 These functions might be defined over only a suitable subset of K , such 
as the set of all non-negative vectors. 



exists an w-only policy a*(t) that yields the following for all 

k€{l,...,K},m£ {1,...,M}: 
{-i 

-£]E {&*(«(*),"(*))} = E{8 fc (a*(t),w(t))} 

T=0 
t-1 

-^E^(a(t),w(t))} = E{y k (a*(t),Lo(t))} 



T=0 
t-1 



i5^E{£ ro (a(t),a;(t))} = E{x m (a*(t),cj(t))} 

Proof: (Lemma |2]i For a given slot t > 0, run the 
ch(t) policy and generate random quantities [ui, a] as follows: 
Uniformly pick a time T £ {0, 1, . . . , t — 1}, and define 
[<D, a] = [w(r), a(T)], being the network state observed at the 
randomly chosen time T and the corresponding network action 
a(T). Because u>(t) is stationary, it follows that uj has the 
stationary distribution 7r(w). Now define the w-only policy 
a*(t) to choose uj £ A u (t) according to the conditional 
distribution of a given ui (generated from the joint distribution 
of [u, a]). It follows that the expectations of bk(-), Vk{-), i m (-) 
under this w-only policy a* (t) are as given in the statement 
of the lemma. □ 

Proof: (Theorem[6]l Fix e > 0, and suppose a(t) is a policy 
that respects the queue backlog constraints (fl9l)-(l20ll. satisfies 
the feasibility constraints (f23b-(f25T>. and such that: 

Um sup f(x(t)) < f opt + e/2 

t— >oo 

Then E {Q k (t)/t} -4- for all k € {1, . . . , K}, and there is a 
slot t* > such that: 

E{Q fc (i*)/f}<e Vk£{l,...,K} (29) 

gi(x(t*))<e VZG{l,...,i} (30) 

1 

- ^E{a fc (r)} > A fc -e Vfe G {1, . . . , K] (31) 

f(x(t)) < fopt + e (32) 

where (EB holds by (dUl. By (ffT) we have for all k £ 
{l,...,K}: 

t*-i 

^2[dk(r)+y k (T)-b k (r)} = Q k (t*)-Q k (0) 

T = 

< Qk{t*) 

Taking expectations, dividing by t*, and using j29l yields: 



t'-i 



±-J2 E Wk(T)+y k (T)~bk(T)}<e 



(33) 



The above holds for all k £ {1, . . . , K}. Using Lemma [2] we 
know there must be an w-only policy a*(t) that satisfies for 
all l£{l,...,L} and all fc G {1, . . . , K}; 

gi (E{x(a*(t),iu(t))}) < e 
\k+E{y k (a*(t),Lu(t))-b k (a*(t),Lo(t))} < 2e 

/(*(*)) < r pt + e 

Redefining e' = 2e proves the result. □ 



x 



We note that if uj(t) is defined by a periodic Markov chain, 
then it can be made stationary by randomizing over the period. 
In the case when oj(t) takes values in a finite set fi, we 
can prove the same result without the stationary assumption 
[16][10]. 



VII. The Capacity Region 



Now define A as the the set of all non-negative rate vectors 
A = (Xi, . . . , Xk) such that for all e > 0, there exists an 
w-only policy such that the constraints d26ll-(l27|i of Theorem 
[6] hold. It can be shown that this set A is a closed set. By 
Theorem[6l we know that A £ A is necessary for the existence 
of an algorithm that makes all queues Q k (t) mean rate stable 
and that satisfies the gi(-) constraints ( f23l ). Because, under 
some mild technical assumptions, mean rate stability is the 
weakest form of stability, it follows that the constraint A € A is 
also a necessary condition for stabilizing the network (subject 
to the gi(-) constraints) under either rate stability, steady state 
stability, or strong stability. 

We now show that the set A is the network capacity region, 
in the sense that, under some mild additional assumptions 
on the processes a k (t) and ui(t), it is possible to make all 
queues Q k (t) strongly stable whenever A is an interior point 
of A. Because the technical assumptions we introduce will also 
imply that strong stability is the strongest stability definition, 
it follows that the same algorithm that makes all queues Qk(t) 
strongly stable also makes them rate stable, mean rate stable, 
and steady state stable. For simplicity of exposition, we treat 
the case when the functions f(x), gi(x) are linear or affine 
(the case of convex functions is treated in [7][10][12][21]). 
Note that E{f(X)} = f(E{X}) for any linear or affine 
function and any random vector X. 



A. The Decaying Memory Property 

Suppose that has stationary distribution it(uj) as before, 
and that arrival processes a&(i) have rates X k that satisfy 
< fT~8T >. Define H(t) as the history of the system over slots 
t € {0, 1, . . . , t — 1}, consisting of the initial queue states 
(5^(0) and all w(r), a(r) values over this interval. We say that 
the processes dfc(t) and u>(t) together with the functions b k (-), 
2/fc(*)> 9l{&('))> have the decaying memory property if for any 
w-only policy a* (t) and any 5 > 0, there is an integer T > 
(which may depend on 5 and a*(t)) such that for all slots 
to > 0, all possible values of H(to), and all k € {1, ... , K}, 



I € {1, . . . , L} we have: 

, to+T-l 

- J2 E{a fc (r)+y fc (a*(r), W (r))|i/(t )} 

T = t 

, to+T-1 

-- ^{bk(a*(r)Mr))\H(to)}<X k 

T = t 

+E, {fo(a*(t),w(t)) - b k (a*(t),u(t))}+5 (34) 

, to+T-l 

- J2 E{«w(sfe(a(r),a;(r)))|ff(to)} 

T = t 

-lM0i(a(a*(t),w(t)))}<« (35) 

, to+T-l 

- E{/(i(a(r), W (r)))|ff(*o)} 

T = to 

-E^ifixia^t),^)))} <S (36) 

where {•} represents an expectation over the stationary 
distribution tt(lu) for u>(t). Intuitively, the decaying memory 
property says that the affects of past history decay over T 
slots, so that all conditional time average expectations over this 
interval are within S of their stationary values. This property 
can be shown to hold when u>(t) and Ofe(t) are driven by a 
finite state irreducible (possibly not aperiodic) Markov chain, 
and when the conditional expectation of all processes is finite 
given the current state. 

B. Second Moment Boundedness Assumptions 

We assume that for all t and all (possibly randomized) 
control actions a(t) € A<>(t)> tne second moment of the 
processes are bounded, so that there is a finite constant a 2 
such that: 

E{y k {a(t),Li(t)) 2 } < a 2 

E{b k (a(t),u(t)) 2 } < a 2 

E{ gi (x(a{t),u(t))) 2 } < a 2 
E{a k (t) 2 } < a 2 

Note that these second moment assumptions also ensure first 
moments are bounded. Finally, we assume the first moment of 
f(x(t)) is bounded by finite constants f m i n and f max , so that 
for all (possibly randomized) control actions a(t) e A u m we 
have: 

fmin < E {f{x(a(t), w(t)))} < f max (37) 

C. Lyapunov Drift 

We use the framework of [7] [ 10] [ 1 1] to design a policy to 
solve the optimization problem (I22l i-(|25]|. To this end, for each 
inequality constraint ( 1231 define a virtual queue Zi(t) that is 
initially empty and that has update equation: 

Zi(t + 1) = mnx[Zi(t) + gi (x(t)),0] (38) 

where x(t) = x(a(t) t u)(t)). The actual queues Q k (t) are 
assumed to satisfy il7\ . 
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Define &(t)=[Q(t), Z(t)} as a composite vector of all 
actual and virtual queues. Define a Lyapunov function L(&(t)) 
as follows: 

K L 

L(0(t))A-^Q fcW 2 + -^Z i (t) 2 

k=l 1=1 

For a given integer T > 0, define the T-step conditional 
Lyapunov drift Ax(&(t)) as follows^ 

A T (0(i))4E {L(&(t + T)) - L(®(t))\&(t)} (39) 

Lemma 3: For any control policy and for any parameter 
V > 0, A T (0(i)) satisfies: 

t+T-l 

A T (0(i)) + V J2 E{/(x(r))|©(t)}< 

t+T-l 

T 2 E{B|0(i)} + ^ £ E{/(x(t))|0(*)} 

+ E E K{Q k (t)[a k (r)+y k (T)-b k (r)}\@(t)} 

k=l r=t 
L t+T-l 

+ E i Z i(t)9l(x(r))\®(t)} (40) 

1 = 1 T = t 

where B is a random variable that satisfies: 
E jsj < B 

where B is a finite constant that depends on the worst case 
second moment bounds of the a k (r), b k (r), y k (r), gi{x(r)) 
processes, as described in more detail in the proof. 

Proof: See Appendix G. □ 
The parameter V > will determine a performance -backlog 
tradeoff, as in [10] [7] [11] [12]. 

D. The Drift-Plus-Penalty Algorithm 

Consider now the following algorithm, defined in terms of 
given positive parameters C > and V > 0. Every slot t, 
observe the current ui(t) and the current actual and virtual 
queues Q k (t), Zi(t), and choose a control action a(t) £ A u (t) 
that comes within C of minimizing the following expression: 

L 

1=1 

K 

+ 52Q k (t)[y k {a(t),uj(t))-b k {a(t),u}{t))\ 

k=l 

This algorithm is designed to come within an additive constant 
C of minimizing the right hand side of d40b over all actions 
a(t) £ -A w (t) tna t can be made, given the current queue states 
0(i). We call such a policy a C -approximation. Note that a 
0-approximation is one that achieves the exact infimum on the 
right hand side of (1401 . The notion of a C-approximation is 

4 Strictly speaking, better notation is Ar(6((),t), although we use the 
simpler notation Ay (0(i)) as a formal representation of the right hand side 
of (39). 



introduced in case the infimum cannot be achieved, or when 
the infimum is difficult to achieve exactly. 

Lemma 4: Suppose we use a C-approximation every slot. 
Then for any time t, any integer T > 0, and any &(t), we 
have: 

t+T-l 

A T (&(t)) + V E{/(x(r))|8(*)}< 

T = t 

CT + T 2 E |B|0(t)| 

t+T-l 

+T(T-i)E{i)\e(t)} + v Y E{/(**(T))|e(t)} 

T = l 

K t+T-l 

+ £>W E Mak(T)+y* k (T)-bUr)\®(t)} 

k=l T=t 
L t+T-l 

+ E Z 'W E E{Sl(*»)|©(t)} (41) 

1=1 T = t 

where x*(t), y k {T), b\{r) are values that correspond to any 
alternative policy for choosing q*(t) € A u ( T y. 

x*(t) A x(a*( T ),w(r)) 
I,J(r) A fo(o*(r), W (T)) 
6JJ(r) A S fc (a*(T),w(r)) 

and where £> is a random variable that satisfies: 

E j£>j < L> 

where L> is a finite constant related to the worst case second 
moments of a k (t), b k {t), y k (t), gi(x(t)), described in more 
detail in the proof. 

Proof: See Appendix F. □ 

E. Algorithm Performance 

In the following theorems, we assume the set A is non- 
empty and has non-empty interior. We say that a non-negative 
rate vector A is interior to A if A g A and if there exists a 
value dmax > such that A + d max e A, where d max = 
(d ma x,d ma x, d max ) is a vector with all entries equal to 
dmax- We also assume the initial condition of the queues 
satisfies E{L(0(O))} < oo@ 

In the case when L = 0, so that there are no <#(•) 
constraints, we only require A to be an interior point of A. 
However, if L > we need a stronger assumption, related to 
a Slater-type condition of static optimization theory [27]. 

Assumption Al : There is a constant d max > and an cj-only 
policy that yields for alH e {1, . . . , L}, fee {1, . . . , K}: 

gi {E{x(a*(t),u(t))}) < -d max /2 (42) 
X k + E ^y k (a*(t),uj(t)) - b k (a*(t),uj 

(*))} < /2 (43) 

It is clear from Theorem [6] that Assumption Al holds 
whenever A + d max € A and when L = 0. This is because 
if A + d max £ A, then for any e > 0, Theorem [6] implies 

5 Note that E{L(©(0))} = if all queues are initially empty with 
probability 1, 
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the existence of an w-only policy a*(t) that satisfies for all 

ke{l,...,K}: 



A fe + d max + E {y k (a*(t),Lo(t)) ~ b k (a*(t), w(t))} 



< e 



and hence we can simply choose e = d max /2 to yield (|43| > 
(note that d42l is irrelevant in the case L = 0). 

Theorem 7: Suppose Assumption Al holds. Suppose we 
use a fixed parameter V > 0, and we implement a C- 
approximation every slot t. Suppose the system satisfies the 
decaying memory property of Section [VII- A| and the bounded- 
ness assumptions of Section [VH-B I Then all actual and virtual 
queues are mean rate stable and so: 

limsup gi(x(t)) < VZe{l,...,£} (44) 

lim E{Q k (t)/t} = Vfc€{l,...,*r} (45) 

Therefore, all desired constraints of problem (l22ii-(|25ll are 
satisfied. Further, all queues are strongly stable and: 



K 



1 * _1 
lim sup - 2_, 

t=0 Lk=l 



X;E{Q*(r)} + »{Z,(r)} 
fc=i 

[C + TB + (T - 1)D + V(f max - f, 



L 

i=i 



< 



.)] 



r/4 



(46) 



where T is a positive integer related to d max and independent 
of V, and -B and D are constants defined in Lemmas [3] and 
|U Further, for any e such that < e < d max /4, there is a 
positive integer T e , independent of V, such that: 



lim sup /(x(r)) 

t— >oo 



where cq is defined: 



< 



£Opt 

c 



c e 
BT f 



D(T e - 1) 



V 



(47) 



The constants T, T t are related to the amount of time required 
for the memory to decay to a suitable proximity to a stationary 
distribution, as defined in the proof. 

Theorem [7] shows that the algorithm makes all queues mean 
rate stable and satisfies all desired constraints for any V > 
(including V = 0). The parameter V is useful because the 
achieved cost can be pushed to its optimal value f opt as V —> 
oo, as shown by fflJ\ - Hence, we can ensure the achieved cost 
is arbitrarily close to the optimum by choosing V suitably 
large. While a larger value of V does not affect the constraints 
(l44l . (|45T >. it turns out that it creates a larger convergence time 
over which time averages are close to meeting their constraints. 
It also affects the average queue backlog sizes, so that the 
average backlog bound is 0(V), as shown in (|46| |. 

In the i.i.d. case, it is known that the difference between 
the achieved cost and the optimal cost f opt is 0(1/ V), which 
establishes an [0(1/V),0(V)] performance-backlog tradeoff 
[10][12][1 1][7]. For the non-i.i.d. case, for a given e > 0, the 
bound d47| i shows that average cost is within 0(1/V) of f°P t + 
Cq6. However, the coefficient multiplier in the 0(1 /V) term 
is linear in T e , representing a "mixing time" required for time 
averages to be within O(e) of their stationary averages. If we 



choose e = 1/V, then this mixing time itself can be a function 
of V (we typically expect T t = 0(log(l/e)) = 0(log(V)) if 
the network events are driven by a finite state Markov chain). 
This would present an [0(\og(V)/V),0(V)] performance- 
delay tradeoff. Such a tradeoff is explicitly shown in [17] [18] 
for particular types of networks, where a worst case backlog 
bound of 0(V) is also derived. Work in [20] treats a mobile 
network with non-ergodic mobility, defines an "instantaneous 
capacity region," and shows (using an analysis similar to i.i.d. 
analysis) that achieved utility is within 0(1/ V) of the sum of 
optimal utilities associated with each instantaneous region. 

Our work [20] also states an extension (without proof) that 
for ergodic mobility, the achieved utility is within B mo bu e /V 
of the optimum, where B mo bu e is a constant associated with 
the timescales of the mobility process, although it does not 
compute this constant. This statement is in Theorem 1 part (f) 
of [20]. The result ( f4Tb above shows the constant B mo bu e can 
be defined for a given e as: 



B,, 



obtlc 



c eV + C + BT e + D(T e - 1) 



Defining e=l/V removes dependence on V in the first term, 
but there is still dependence on V in the terms that are linear 
in T e . Typically, T x iy is 0(log(F)) for systems defined on 
finite state ergodic Markov chains. However, our statement in 
part (f) of [20] claims a constant B mo bu e can be found that is 
independent ofV (yielding an 0(1 /V) distance to optimality, 
rather than an 0(\og(V) /V) distance). This is indeed the case, 
although it requires the ui(t) process to be either i.i.d. over 
slots, or to be defined by an ergodic Markov chain with a 
finite state space@ 

Proof: (Theorem IT) By Assumption Al there is an w-only 
policy a*(t) such that for all k &{!,... ,K}, I £ {1, ... , L}: 



E{ m (x(a*(t),oj(t)))} < -d max /2 (48) 
A fc +E{fo(a*(t),w(t)) - b k (a*(t),u(t))} < -d max /2 (49) 



where we have used the fact that <?;(•) is linear or affine to 
pass the expectation through this function in ((48). 

Fix 8=d max /A, and choose a frame size T > that satisfies 
the decaying memory properties d34li-(|36i> for this 5 and this 
w-only policy a*(t). The value of T depends on 5, and so 
we could write Tg, although we use T below for notational 
simplicity (we note the dependence on S again when needed). 



6 We note that our statement in part (f) of [20] says that for ergodic mobility, 
a constant -B mo j,;; e can be found that is independent of V. This should have 
been stated more precisely for mobility defined by a "finite state ergodic 
Markov chain." We regret this ambiguity. The full proof of this result is due 
to Longbo Huang and uses a variable frame drift analysis that is slightly 
different from the T-slot drift analysis suggested in the proof outline in [20] . 
The details are in preparation. 
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By Lemma |4] we have: 



t+T-l 



A T (®(t)) + V J2 E{f(x(r))\e(t)}<CT 

T=t 

+T 2 E |s|0(t)| + T(T - 1)E |£)|0(t)| 

t+T-l 

+^ X! e{/(^(t))i©(*)} 



T = t 

K t+T-l 



Y,Qk{t) E E{o fc (r)+i,iE(r)-6X(r)|e(t)} 

fc=l T=t 

L t+T-l 

5> { (t) E E{ 5l (x*(r))|0(t)} (50) 



z=i 



By the decaying memory property, the conditional expecta- 
tions given 0(t) are within 5 = d max /A of their stationary 
averages, and so (by using (|48l)-(|49ll): 



t+T-l 



A T (e(t)) + V E E{/(a J (r))|©(t)}<CT 
+T 2 E {S|0(i)} + T(T - 1)E {£]©(*)} + VTf max 

K L 

- J2 Qk(t)Td max /4: - J2 Z^Td^/A (51) 



k=i 

Rearranging terms yields: 



1=1 



M®(*)) 



TcL 



A" 



E^w + E z 'W 



U-l 



< 



CT + T 2 E |s|0(i)| + T(T - 1)E |£>|0(i)| 

~^~VT(f max fmin) 

Taking expectations of both sides and using the law of iterated 
expectations yields: 



;{L(0(* + T))}-E{£(0(t))} 
lfe=l 1=1 ) 



<CT + T 2 B + T(T - l)D + VT(f max - f mm ) 

The above holds for all slots t > 0. Define <i = to + VT for 
i £ {0, 1,2, . . .}, where t e {0, 1, . . . ,T- 1}. We thus have: 



E{L(0(t l+1 ))}-E{L(0(i I ))} 



A" 



.fe=i 



/=i 



< CT + T 2 B + T(T - l)D + VT(f„ 



fmin) 



Summing over i e {0, . . . , J — 1} yields: 

E{L(0(t J ))}-E{L(0(t o ))} 



j-i 

E 

i=0 



A 



EE{o fc (^)} + E E {^^)> 



.fc=i 



/=i 



< J[CT + T 2 B + T(T - l)D + VT(f„ 



fmin)] 



Rearranging terms and using the fact that E{L(&(tj))} > 
yields: 



J-i 

E 

i=0 



A 



EE{Qfe(ti)} + EE{^(ii)} 



,fc=i 



;=i 



< 



J[CT + ST 2 + T(T - l)D + VT(f max - f min )} 



E{L(0(t o ))} 



Summing over all to g {0, 1, . . . , T — 1} and dividing by JT 
yields: 

" K L 

E E {<9fe(r)} + E E (r)} 



JT-l 

— F 



,fc=i 



;=i 



< 



[C + TB + (T - l)D + V(f max - f min )] 



^rnax / 4 
1 T_1 

to— U 



E{L(0(t o ))} 



The above holds for all positive integers J. Taking a limsup 
as J — > oo yields 



1 4-1 
lim sup — 



T=0 



A 



EE{Q fc (r)} + EE{^(r)} 



,fc=i 



< 



[C + TB + (T- 1)D + F(/ ma:c - f mm )} 

dmax I 4 

This proves d46i >. and hence proves strong stability of all 
queues. By Theorem|4] since the second moments of the arrival 
and service processes for all queues are bounded, we know 
mean rate stability also holds for all queues. Because queues 
Zi(t) are mean rate stable, and these queues have update 
equation ([38}, we know from Theorem [3] that: 

1 t_1 

limsup -E E {g;(z(T))} <0 

1 T =0 

Passing the expectation through the linear function gi(x) 
proves (144b . 

It remains only to prove d47| ). Fix e > 0, and assume that 
e < d max /A. Note that Theorem [6] implies the existence of an 
w-only algorithm a'(t) that satisfies: 

E{ gi (x(a'(t),Lu(t)))} <e (52) 

A fc +E{$fc(a / (t),w(t))-S fc (a*(t),w(t))} <e (53) 

E{f(x{ a '{t),uj{t)))}<r t + ^ (54) 

where we have used the fact that /(•), <?;(•) are linear or 
affine to pass expectations through them. Now define an w- 
only policy a*(t) as follows: 

a* (t) with probability 9 
a' (t) with probability 1—9 



«*(*) 



7 While the above only samples the time average expectation over slots that 
are multiples of T, it is easy to see that the limsup time average over all 
slots is the same, as the queue cannot change by much over the course of T 
slots. 
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where a*(t) is the algorithm of Assumption Al, and where 9 
is defined: 

9±Ae/d max (55) 

Note that 9 is a valid probability because < e < d max /4. 
Therefore, under policy a*(t) we have (combining d52ll-(l54li 
and <@3-<g3]>): 

E{ 5j (x(a*(t),w(t)))} < 

(1 - 6)e - 9d max /2 < -e (56) 

A fe + E {fe(a*(f), w(t)) - S fc (a*(t),w(*))} 

< (1 - 0)e - #d ma;c /2 < -e (57) 
E{/(*(a*(t),w(*)))} < (1 - + 9f max (58) 

where we have used the fact that 9 = 4e/d max to conclude 
that: 

9d max /2 = 2e 

Now fix 5 = e, and define T e as the value that satisfies the 
decaying memory properties (|34]i-([36]> for this 5 and for the 
w-only policy a*(t). By Lemma [4] 

t+T.-l 

A T£ (0(t)) + F ^ E{/(*(r))|e(t)}<CT 

T=t 

+T e 2 E j_B|0(i) j + T £ (T e — 1)E |z>|0(t)| 

t+T e -l 

+F ]T E{/(x*(r))|©(<)} 

r = t 

if t+T e -l 

+ £<?*(*) E E{o fc (r)+»J(r)-6j(r)|e(t)} 

fc=l T=t 
L t+Te-l 

+ E Z 'W E E{5l(**M)|e(t)} (59) 

i = l T=t 

Noting that the decaying memory property ensures the above 
conditional expectations are within 8 = e of their stationary 
averages d56ll-(l58ll. we have: 

t+Te-l 

A Te (&(t)) + V J2 !{/(s(r))|©(i)}< 

T=t 

CT + T e 2 E |j3|©(t)| + T e (T e - 1)E |£>|©(t)| 
+VT e (r t + 6f max + e) 

Taking expectations gives: 

E{L{&(t + T t ))}-E{L(&{t))} 

t+Te-l 

+V J2 E if( x ( T ))} <CT + T?B + T £ (T e -l)D 

T=t 

+VT e (f opt +9f max + e) 

As before, we substitute t = t j for i g {0,1,2,...,} for some 
value i € {0, 1, . . .T— 1}, and sum over i G {0, 1, . . . , J— 1} 



and t € {0, 1, ... ,T - 1} to get: 

v JTe " 1 

— ^{f{x{T))}<[C + T c B+(T e -l)D} 

£ T = 

+y(/ o P t + 0fmax + e) + E {i(e(to))} 

c to=0 

Dividing by V and taking a limit as J — > oo yields: 

limsup/(x(r)) < r pt + Of max + e 

t— yoo 

C + T t B+ (T £ - l)D 
+ V 

where we have used the fact that f(x) is linear or affine to pass 
the time average expectation through it. Using 9 = Ae/d max 
proves (|47] i. □ 

VIII. Exercises 

Exercise 1: (Inequality comparison) Let Q(t) satisfy (Q]) 
with server process b(t) and arrival process a(t). Let Q(t) 
be another queueing system with the same server process b(t) 
but with an arrival process a(t) = ait) +z(t), where z(t) > 
for all t G {0, 1,2,.. .}. Assuming that Q(0) = Q(0), prove 
that Q(t) < Q(t) for all t G {0, 1,2,.. .}. 

Exercise 2: (Proving sufficiency for Theorem |2^) Let Q(t) 
satisfy ((TJ with arrival and server processes with well defined 
time averages a av and b av . Suppose that a av < b av . Fix e > 0, 
and define Q e (t) as a queue with Q e (0) — Q(0), and with the 
same server process b(t) but with an arrival process a(t) = 
a(t) + [b av - a a v) + e for all t. 

a) Compute the time average of a(t). 

b) Assuming the result of Theorem I2J3, compute 
limt-voo Q e {t)/t. 

c) Use the result of part (b) and Exercise Q] to prove that 
Q(t) is rate stable. 

Exercise 3: (Proof of Theorem I2J?) Let Q(t) be a queue 
that satisfies ([TJ. Assume time averages of a(t) and b(t) are 
given by finite constants a av and b av , respectively. 

a) Use the following equation to prove that 
lim^oo a(t)/t = with probability 1: 

b) Suppose that 6(tj) < b(ti) for some slot ti. Use (UJ to 
compute Q(ti + 1). 

c) Use part (b) to show that if 6(tj) < b(ti), then: 

ti 

a(U)>Q(0) + J2[a( T )-b(T)] 

T = 

Conclude that if b(ti) < b(ti) for an infinite number of slots 
U, then a av < b av . 

d) Use part (c) to conclude that if a av > b av , there is some 
slot t* > such that for all t > t* we have: 

t-i 

QW = Q(0+EHt)-6(t)] 

r=t* 
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Use this to prove the result of Theorem [2J3. 

Exercise 4: (Strong stability implies steady state stability) 
Prove that strong stability implies steady state stability using 
the fact that E{Q(t)} > MPr[Q{r) > M}. 

Appendix A — Proof of TheoremQ] 

Here we prove Theorem Q] Note that rate stability implies 
that for any 8 > 0, we have|^ 



lim Pr[Q(t)/t > 8} = 



(60) 



For a given 8 > 0, define the event £ t ={Q(t)/t > 8}, so that 
lim^oo Pr[£ t ] = 0. Define £%±{Q{t)/t < 8}. 

Lemma 5: If Q(t) is non-negative and satisfies ( |60l >, and if: 

UmE{Q(t)/t\£ t }Pr[£ t } = (61) 

t— J-oo 

then Q(t) is mean rate stable. 

Proof: We have for a given 6 > 0: 

0<E{Q(t)/t} = E{Q(t)/t\£?}Pr[£Z] 
+E{Q(t)/t\£ t }Pr[£ t ] 
< 8 + E{Q(t)/t\£ t }Pr[£ t ] 

Taking a lim sup of both sides and using d6Tb yields: 

< limsupE{Q(i)/t} < 8 

This holds for all 8 > 0. Thus, lim t _ ) . 0O E{Q(t)/t} = 0, 
proving mean rate stability. □ 

Thus, to prove TheoremQ] it suffices to prove that (|6TT > holds 
under the assumptions of parts (a) and (b) of the theorem. To 
this end, we have a preliminary lemma. 

Lemma 6: If X is a non-negative random variable such that 
E |X 1+e } < 00 for some value e > 0, then for any event £ 
with a well defined probability Pr[£], we have: 

E{X\£}Pr[£] <E{X 1+c } 1/{1+e) Pr[£] e/il+e) 
Proof: If Pr[£] = 0, then the result is obvious. Suppose 
now that Pr[£] > 0. We have: 

E{X 1+e } = E{X 1+e \£} Pr[£]+E{X 1+e \£ c } Pr[£ c ] 
> E{X 1+t \£}Pr[£] 

Therefore: 



E{X 1+e \£} < 



Pr[£] 



m = 



Thus: 



Hence: 



for x > 0, we have: 



E{X\£} 1+e < E{X 1+£ \£ } 
E{X 1+e } 



E{X\£} 1+£ < 



Pr[£] 



E{X\£} < 



'E{X 1+e }' 
Pr[£] 



8 In fact, the result of parts (a) and (b) of Theorem [T] hold equally if the 
assumption that Q(t) is rate stable is replaced by the weaker assumption (60). 



Multiplying both sides by Pr[£] proves the result. 

We now prove part (a) of Theorem Q] 

Proof: (Theorem Q2 a )) F° r simplicity assume that Q(0) 
0. Note that: 



□ 



t-i 



«M<i5>(r) + 6-(r)] 



r=0 



Define X(r)4a(r) + 6"(r). Thus: 



Q(t) / 1 



< 



(62) 



Now suppose there are constants e > 0, C > such that: 

E{X(r) 1+e } < C for all r (63) 
Fix 5 > and define the event £ t ={Q(t)/t > 8}. Thus: 

E{^|£ t }pr[£y < i^E{X(r)|f t }Pr[f t ] (64) 
< I^cV(i+e)p r [£ t ]e/(i+e) 



r=0 



C l/(l+e)p r[< P t] e/(l+e) 



where the first inequality follows by (1621 and the second 
inequality uses Lemma [6] together with (l63l . Taking a limit 
of the above and using the fact that Pr[<?t] — > yields: 

lim E{Q(t)/t\£ t }Pr[£ t ] =0 

t— >oo 

and therefore Q(t) is mean rate stable by Lemma |5] □ 
To prove part (b) of Theorem Q] we need another prelimi- 
nary lemma. 

Lemma 7: If X is a non-negative random variable and £ is 
any event with a well defined probability Pr[£], then for any 
i>0we have: 

E {X\£} Pr[£] < E {X\X > x} Pr[X > x] + xPr[£] 
Proof: Fix a value x > 0. Define indicator functions 
and l{x>x} as follows: 

1 if event £ is true 

otherwise 

1 if X > x 
{x>x} = \ q otherwise 



1 



However, by Jensen's inequality for the convex function Define 1 



{X<x} = ^ " ^{X>x}- 

Xl£ = Xl £ (l[x>x} + l{X<x}) 

= Xl £ l{x>x} + Xl £ l{ X <x} 

Therefore: 



Then: 



E {Xl £ } < E {X1 {X>X }} + xPr[£] 



Thus: 



E{X\£}Pr[£] < E{X\X > x} Pr[X > x]+xPr[£] 



□ 
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Proof: (Theorem Q2b)) Fix 5 > and define the event 
£ f A{Q(t)/t > 6}. From dH we have: 

E {^|£ t }pr[£y < j^E{X(T)\£t}Pr[£t](65) 

where we recall that X(r)=a(r) + b~(r). Now fix any 
(arbitrarily large) x > 0. From Lemma [7] we have: 

E{X(r)\£ t }Pr[£ t ] 

< E{X(t)\X > x}Pr[X(r) > x] + xPr[£ t ] 

< E {Y \Y > x} Pr[Y > x] + xPr[£ t ] 

where the final equality has used the assumption about the 
random variable Y in Theorem Q] part (b). Plugging this into 
(|65T > yields: 

E | \£ t | Pr[£ t ] < E {Y\Y > x} Pr[Y > x] 

Because Pr[£ t ] —> as t — > oo, its time average also 
converges to 0. Thus, taking a limsup of both sides of the 
above inequality as t — > oo yields: 

'Q(<), 



lim sup E 



\£t > Pr[£t] < E > x} Pr[Y > x] 



The above holds for all x > 0. The fact that E{F} < 
oo ensures that the right hand side of the above inequality 
vanishes as x — > oo. Taking a limit as x — >• oo thus proves: 



lim sup E <j ^ Pr[£ t ] = 



and hence: 



KmE(^|fJPr[f t ] = 

This together with Lemma |5] proves that Q(t) is mean rate 
stable. □ 

Appendix B — Proof of TheoremBJc) 

Here we prove part (c) of Theorem H] The proof is similar 
to our previous proof in [28]. 

Proof: (Theorem |4] part (c)) Suppose there is a finite 
constant C > such that E{b(t) - a(t)} < C for all t, and 
that Q(t) is not mean rate stable. It follows that there is an 
e > such that E {Qitkj/tk} > e f° r an infinite collection of 
times tk- For any t% and any t > tk we have by dU): 



Q(t) > Q(tfc) 



t-i 

E 

T=tfc 



[b(r)-a(r)] 



Thus, for any i > tk we have: 

E{g(t)}>ei fe -(i-t fc )C 

Now fix any (arbitrarily large) value M > 0. Then for 
sufficiently large k we have etk > M, and etk — (t—tk)C > M 
whenever: 

(C + e)t fc - M 



t* < t < 



C 



(1 + e/C)t k - M/C (66) 



Hence, E{Q(t)} > M whenever (|66l ) holds. Define fj. as: 

t k ±l(l+e/C)tk-M/C\ 

The number of slots in the interval tk < t < tk given by 
is at least: 

i k -t k + l>et k /C-M/C 



It follows that 

i 



tk 



1 j2 E iQ( T )}> M 

+ T=0 



ct fc /C - M/C 



(1 + e/C)t k - M/C + 1 



Taking a limsup as k —> oo and noting that lim^oo = oo 
yields: 



l imsu p^^^E{Q(r)}> - 



M e/C 



This holds for arbitrarily large M. Hence, taking a limit as 
M — » oo yields: 

Km sup , V E {Q(r)} > oo 

and thus is not strongly stable. It follows that strongly 
stable implies mean rate stable. 

A similar proof can be done for the case when 
E{a(t) + b~(t)} < C for all t. This can be shown by 
observing that for any t < t k : 

t-i 

Q(t)>Q{t k )- X>( r ) + M T )1 



and hence: 



E{Q(t)}>et k -C(t-t k ) 



□ 



Appendix C — Proof of TheoremBJb) 

Here we prove Theorem 5|b), which shows that strong sta- 
bility implies rate stability if certain boundedness assumptions 
are satisfied. Suppose Q(t) has dynamics given by £[), and that 
there is a finite constant C > such that with probability 1, 
we have: 



b(t)-a(t) <CVte {0,1,2,...} 



(67) 



For simplicity, we assume the condition d67l > holds determin- 
istically (so that we can neglect writing "with probability 
1.") Suppose Q(t) is strongly stable. We want to show that 
lim^oo Q(t)/t = with probability 1. 

We prove the result through several preliminary lemmas, 
presented below. 

Lemma 8: If Q(t) is strongly stable and if there is a 
finite constant C > such that (|67| > holds for all t, then 
E{Q(t)/t] < 0(1/ Vi)- Specifically, there exists a finite 
constant D > and a positive timeslot to such that 

(68) 



(69) 



E{Q(t)/t} < D/Vt for all i>i D 
Hence, for all t > tp and all e > we have: 

Pr[Q(t)/t > c/4] < AD/(eV~t) 
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Proof: Because Q(t) is strongly stable, there is a finite 
constant B > such that: 



1 

limsup-^E{Q(Y)} < B < 



T = 



Then for large t (all t > t* for some t*), we have: 



\Y,¥,{Q{t)}<B 



T = 



Suppose now that for any finite D > 0, there exist arbitrarily 
large times ti such that E {Q(ij)/ti} > D j %/t[. We shall reach 
a contradiction. For U > t* we have: 



1 2t;-l 1 2tj-l f 

B>-£E{Q(t)} > ,E E 

1 t=0 r=t< 



1 2 tl -l f 
1 — ^ 



> 



Q(r) 

2 ^ — 



(70) 



Because ( |67l > holds, we know that for all r > ^: 

Q(r) > Q{U) ~ C(t - t t ) 

Further, E{Q(U)/U} > D/y/U and so E{Q(i;)} > Dy^ 
We thus have for all r G {t i; . . . , 2i, - 1}): 

E{Q(t)} D^U-C{t-U) 

T T 

Now assume that fj is large, so that 2tj — 1 > ti + 
[D^/U/{2C)\ >U + £>V*I/(4C). Note that if: 

r€{*i,...,*i+Ll>Vti/(2C)J} 

then t — ti < Dy/ti/ (2C) and we know: 

E{Q(r)} D^U-CjT-U) 



> 



D^/ti-C{r-ti) 



2ti 

> D/(4VU) 

It follows that: 

Therefore, for large ti, from (|70l we have: 

D 2 



B > 



32C 

The above inequality must hold for all D > 0. This clearly 
does not hold for D > y/32BC, yielding a contradiction and 
hence proving the result d68l >. 

To prove d69| i, we use the fact that d68l l holds to get for any 
time t > to- 

D/Vt >E{Q(t)/t} > {e/A)Pr[Q{t)/t > e/4] 



Dividing the above by e/4 yields ((69) ■ □ 
Now again suppose a(t) and b(t) satisfies (loTl i for all t. Fix 
e > and define a constant a > as follows: 

e 1 

2C' 2 



Fix an integer time to sucn that to > l/a, and define the 
following sequence for i g {0, 1,2,.. .}: 

fj+i = U + [aU\ 

Note that for all i we have ti+\ > ti (because ati > 1). Let 
the set of times in the interval {ti, . . . , U + [ati\ — 1} denote 
the ith frame. 

Lemma 9: If Q(t)/t > e for any time t > to in the ith 
frame (for some i £ {0, 1,2,.. .}), then > e/4. 

Proof: Fix e > 0, and suppose Q(t)/t > e, where t > to 
and t is in the ith frame for some i £ {0, 1,2,.. .}. Then: 



Hence: 



However: 



U < t < t i+ i <U + ati 



ti+i -t<ati 



(71) 
(72) 



Q(U+i) > Q(t) - C(t l+1 - 1) 

Therefore (because Q(t)/t > e) we have: 

Q{U+i) > et-C(t i+ i-t) 

> eti - C{t l+ i - t) 

> eti — Cati 

> Ue/2 



(73) 
(74) 
(75) 



where ( 1731 follows because t > ti, d74l follows from d72~t . 
and (|75] l follows because a < e/(2C) (by definition of a). 
Thus: 

Qitj+i) e / U 
ti+i 2 

However, from (1711 we have: 

ti+i ~ U < ati 

and so: 

1 - (U/U +1 ) < a{U/t i+ i) < a 



Thus: 



H+l 



> I- a > 1/2 



where the last inequality follows because a < 1/2 (by 
definition of a). Using this in ft75[ yields: 

^±il > (J±-) e/2 > e/4 
ti+i \ti+\ J 

This proves the result. □ 
Lemma 10: For each frame i £ {0, 1,2,.. .} we have: 

Pr[Q(t)/t > e for some t in the ith frame] 

< Pr[Q{t i+1 )/t i+1 > e/4] 
Proof: This follows immediately from the previous Lemma. 

□ 

The remainder of the proof is similar to the standard proof 
of the strong law of large numbers (see, for example, [29]). 
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We have for any to that starts the frames: 



Pr 



limsup Q(t)/t > e 

t— >oo 



< Pr 



sup Q(t)/t > e 



t>t 



< Pr[Q(r)/T > e for some r in the ith frame] 



i=0 



< 



E 

i=0 



Pr[Q{t l+1 )/t l+1 > e/4] 



(76) 



However, ti is an exponentially growing sequence (note that 
ti+i > (1 + a)ti — 1), and so we are sampling Q(t)/t at 
exponentially increasing times. However, if Q(t) is strongly 
stable, from Lemma we know that Pr[Q(t)/t > e/4] < 
0{l/y/t). Hence, the final sum in ( |76] | is summable and goes 
to zero as to —> oo. This proves that if Q(t) is strongly stable, 
then for any e > we know: 



Pr 



limsup Q(t)/t > e 

t— >oo 



= 



Because this holds for all e > 0, it must be the case that 
Q(t)/t — > with probability 1 (and so the queue is rate stable). 

This proof considers the case when b(t) — a(t) < C for all 
t. The other case when a(t) + b~(t) < C for all t is proven 
similarly and is omitted for brevity. 

Appendix D — Proof of Theorem[5] 

Here we prove Theorem[5] Suppose there is a finite constant 
C such that a(t) + b~(t) < C with probability 1 for all t. 
For simplicity, we assume that Q(0) — 0, and that a(t) + 
b~ (t) < C deterministically (so that we do not need to repeat 
the phrase "with probability 1"). Suppose that Q(t) is not 
mean rate stable. We show that it is not steady state stable. 

Because Q(t) is not mean rate stable, there must be an 
e > and an infinite collection of increasing times tk such that 
E{Q{t k )/t k } > e for all k £ {1,2,.. .} and Km*-**, tk = oo. 
Now fix an (arbitrarily large) value M. We have for any time 
t<t k : 

t k -i 

Q{t k ) < Q(t) + H T ) + b ~( T )} < Q(t) + c (tk - 1) 

T=t 

Thus: 

E {Q(t)} > E{Q(t k )} - C{t k -t)> et k - C{t k - t) (77) 

On the other hand, for t < t k we have: 

E{Q(<)} < MPr[Q(t)<M] 

+E{Q(t)\Q{t) > M}Pr[Q(t) > M] 

< M+ CtPr[Q(t) > M] 

< M + Ct k Pr[Q{t) > M] 

where we have used the fact that Q(t) < Ct < Ct k (because 
Q(0) = 0, the queue increases by at most C on each slot, and 
t < t k ). Thus: 

E{Q(t)} <M + Ct k Pr[Q(t) > M] 



Combining this with d77l > yields: 

M + Ct k Pr[Q(t) > M] > et k - C(t k - t) 
Therefore, for all t < t k we have: 

Prm> > M, > '"- C ^-"- M (78) 

Ut k 



Now suppose that: 

< {t k - t) < (e/2C)t k 
It follows from (|79j that: 

C{t k -t)< {e/2)t k 
Using this in (F78b gives: 

[e/2)t k - M 



(79) 



Pr[Q(t) > M) > 



CU 



The above holds for all t that satisfy (l79b . Now assume that k 
is large enough to ensure that (e/2)t k — M > (e/A)t k (this is 
true for sufficiently large k because lim^oo t k — oo. Thus, 
for sufficiently large k, and if j79l holds, the above bound 
becomes: 



Pr[Q{t) > M] > 



AC 



(80) 



From ( |79l , we see that the number of slots t < t k for which 
holds is at least [e/(2C)]t k . Therefore: 



tk 



T=0 

It follows that 



— Y Pr[Q(r) > M] > -±- (—) ( % 
+ l Z^ L^w 1 ~ t k + l\4:CJ \2C 



limsup y Pt[Q{t) > M] > 

k— s-oo tk "F 1 



T = 



e 

8C2 



Thefore, the function g(M) defined by (13[ satisfies for all 

M>0: 



9(M) > 



8C 2 



It follows that: 



lim g(M) > — > 
M->oo aC z 

and hence Q(t) is not steady state stable. 

Appendix E — Additional Sample Path Results 

We begin by reviewing some general results concerning 
expectations of limits of non-negative random processes. 
These can be viewed as probabilistic interpretations of Fatou's 
Lemma and the Lebesgue Dominated Convergence Theorem 
from measure theory (where an expectation can be viewed as 
an integral over an appropriate probability measure). We state 
these without proof (see, for example, [29]). Both lemmas 
below are stated for a general non-negative stochastic process 
X(t) defined over t > (where t can either be an integer 
index or a value in the real number line). 

Lemma 11: (Fatou's Lemma) For any non-negative stochas- 
tic process X(t) for t > 0, we have: 

liminfE{X(»} >E(liminfX(t)} 
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where both both sides of the inequality can be potentially 
infinite. 

Lemma 12: (Lebesgue Dominated Convergence Theorem) 
Suppose that X(t) is a non-negative stochastic process for 
t > 0, and that there is a non-negative random variable Y, 
defined on the same probability space as the process X(t), 
such that X{t) < Y (with probability 1) for all t. Further 
assume that E{F} < oo. Then: 

(a) limsup^E {*(*)} < E {lim sup^ X(t)} < 
E{Y} 

(b) In addition to the assumption that X(t) < Y with 
probability 1, if X(t) converges to a non-negative random 
variable X with probability 1, then the limit of E{X(i)} is 
well defined, and: 

iim E{X(t)} = E{X} 

t— >oo 

A. Applications to Queue Sample Path Analysis 

Lemma 13: Let Q(t) be a general non-negative random 
process defined over t £ {0,1,2,...}. Suppose that the 
following limit is well defined as a (possibly infinite) random 
variable Q av with probability 1: 

1 

lim -y^Q(r) = Q av with probability 1 (81) 
t->oo t z — ' 

T = 

Then: 

1 

E{Q av } < liminf-V E{Q(r)} 

t— >oo t — ' 

T = 

Therefore, if Q(t) is strongly stable, it must be that Q av < oo 
with probability 1. 

Proof: Define X(t) = \ ^t=o <2( T )' and note that 
liminff^oo X{t) — Q av with probability 1. The result then 
follows as an immediate consequence of Lemma [TT] □ 

Using standard Markov chain theory and renewal theory, 
it can be shown that the limit in dBTt exists as a (possibly 
infinite) random variable Q av with probability 1 whenever 
Q(t) evolves according to a Markov chain such that, with 
probability 1, there is at least one state that is visited infinitely 
often with finite mean recurrence times (not necessarily the 
same state on each sample path realization). This holds even 
if the Markov chain is not irreducible, not aperiodic, and/or 
has an uncountably infinite state space. The converse statement 
does not hold: Note that if the limit ( T8TT > holds with Q av < oo 
with probability 1, this does not always imply that Q(t) is 
strongly stable. The same counter-example from Subsection 
I V-AI can be used to illustrate this. 

Lemma 14: Let Q(t) be a general non-negative random 
process defined over t G {0,1,2,...}. Suppose that for all 
M > 0, the following limit exists as a random variable h(M) 
with probability 1: 

1 t_1 

lim - V UQ(r) > M} = MM) with probability 1 (82) 

T = 

where l{Q(r) > M} is an indicator function that is 1 
whenever Q(t) > M, and zero otherwise. Then the following 



limit for g(M) is well defined: 

t-i 

lim -J2Pr[Q(r)> M] = g(M) 

T = 

Furthermore: 

g(M)=E{h(M)} 

It follows that if Q(t) is steady state stable (so that g(M) -> 
astf-> oo), then: 

lim h(M) = with probability 1 
Proof: The result can be shown by application of Lemma 
M using X(t) = \ J2l~Jo HQir) > M} and Y = 1. □ 
By basic renewal theory and Markov chain theory, it can 
be shown that the limit in d82l holds whenever Q(t) evolves 
according to a Markov chain (possibly non-irreducible, non- 
aperiodic) and such that either the event {Q(t) > M} or its 
complement {Q(t) < M} can be written as the union of a 
finite number of states. 



Appendix F — Proof of Lemma|4] 

Proof: (Lemma |4]l By definition of a C-approximate 
decision, at every slot r £ {t, . . . , t + T — 1} we have: 

VE{f{x{a{r)Mr)))\®{t)} 

L 

i=i 

K 

fc=i 

K 

+ J2^{Qk(r)[M^)^(r))-b k {a(r),Lj(T))]\®(t)} 
k=l 

<C + VE{/(sfe(o*(r),w(T)))|e(i)} 

L 

+ '£E{Z l (T)g l (x(a*(T)MT)))m)} 
i=i 

K 

k=l 

K 

+ ^{Qk(r)M^(r)Mr))-b k (a*(r)Mr))]\®(t)} 

k=l 

where a*(r) is any other (possibly randomized) decision in 
However, we have for all r G {t, . . . , t + T — 1}: 

r-l 

\Z l {r)-Z l {t)\ < J2\9l(^(v))\ 

v=t 
r-l 

\Q k (r) - Q k (t)\ < ^[2/ fc (w)+a fc («)+6 fc ( W )] 

v=t 
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Thus: 



VE{f(x(a(r),u(r)))\@(t)} 



1=1 



J2^{Zi(t)9i(A(a(T),Lo(r)))\&(t)} 

K 
k=l 

^{Qk(t)[ykHT),L0(T))-b k ( a (T)^(T))]\&(t)} 



k=l 



< C + (t - t)2E (i)\®(t)\ 
+VE{f(x(a*(r),cj(r)))\&(t)} 



+ ^TE{Z l (t)g l (x(a*(T),u J (T)))\®(t)} 
1=1 

K 

+ J2^{Qk(t)a k (T)\@(t)} 

k=l 

^{Qk(t)[y k (a%T),Lo(T))-b k (a*(r),Lo(r))}\&(t)} 



k =i 



where 2D is a random variable, with e|_d| < D, where 
D is related (via Cauchy-Schwartz) to the worst case second 
moments of gi(x(t)), a k (t), b k (t), y k {t). A more detailed 
description of D is given at the end of this subsection. 
Summing over r G {t, . . . , t + T — 1} yields: 

(t+T-l \ 

L t+T-l 

+E E ®{Zi(t)gi(H<x(T)MT)))\®(t)} 

1 = 1 T=0 

K t+T-l 

+ E E E{Q fc (t)^(a(r),o;(r))- 

fe=l T=0 

b k {a{r)Mr))]\®{t)} 
CT + T(T-l)E^D\&(t)\ 

+V E E{/(x(a*(r) jW (r)))|0(t)} 



< 

t+T-l 



T = 

L t+T-l 

E E E{Z,(*)9i(!fe(a*(T), W (T)))|e(t)} 

i=l T=0 

if t+T-l 

+ E E E{Q fe (i)[y fe (a*(r),a;(r))- 

fc=l T=0 



b k (a*(r)Mr))]\®(t)} 



The above inequality is an upper bound for the right hand 
side of (1401 1. which proves the result of ( HTT i. □ 



For more details on D and D, we note that 2D can be 
defined to be if r = t, and if t > i it is defined: 

K 

2flA^|( Q *(r) lW (r))+i t ( a *(r), W (r))] x 



fc=i 



r-l 



A" 



fc=l 



1 

— E[ yfe ( w ) + afc ^) + 

v=t 

■ E[^(«( T ), w(t)) + h(a(T), w(t))] 
1 

u=t 

E | 3i (i(«*(T)^(r)))|-^ E 

1=1 v=t 
L r-l 

■Elff«(*(«(r)^(r)))|— -E>0*0>))l 
■Schw 

■{*} 

£> > E E { < w ) + + fefcK, w)) 2 } 



i=l r-; 
By the Cauchy-Schwartz inequality and Jensen's inequality, it 
follows that 2E$d\ < 2D, where D is a finite constant that 
satisfies: 

A" 



k=l 



EE{3/(*(«'») 2 } 



i=i 

where the first term represents the worst case second moment 
of y k { ) + a k (Q) + b k (-) over any decision a' (where the 
expectation is with respect to the stationary distribution 7r(w) 
for oj), and the second term is the worst case second moment 
of 9i (£(■))■ 

Appendix G — Proof of LemmaO 
Proof: (Lemma [3]l From ( TFTT i. it can be shown that (see 
[7]): 

t+T-l 

Qu{t)- E Mr),o 

t+T-l 



,(t + T) < max 



E KM + yfe( T )l 



Using the fact that for Q > 0, fi > 0, a > 

,2 



i (max[Q - (i, 0] + a) 2 < Q ' + ^ + + Q(a - fi) 



yields: 



9=^ s 9 i f + irj : \ i(T ; 



2 ( E [^{r)+y k (T)]\ 
r-i N 

+Qk(*) ( E M t ) + y fc ( T ) ~ 6fc ( r )i 



2 

/t+T-l 



T = t 
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Similarly, from ( TJST l it can be shown that: 

t+T-1 t+T-1 

Z l {t) + 9l(x(r)), £ \9i(*(t)\ 



Zi{t + T) < max 



and so: 



(t+T-1 \ z 

E i^(^w)i) 

t+T-1 

+Zi(t) J2 9l{x(r)) 

T=t 

Combining these, summing, and taking conditional expecta- 
tions yields: 



A r (0(f)) < T 2 E|i3|0(t)| 



K 



t+T-1 



k=l 



Y^Qkit) E E{ak(T)+y k (r)-b k (r)\®(t)} 

L t+T-1 

+ E E{ ffI (*(r))|e(t)} 



i=l r=t 

where B is a random variable that satisfies: 

t+T-1 ^ 



E 



K / t+T- 

M= ^ E 

fe=l ^ \ r=t 
fe=l ^ \ r=t 

+E4p 

i=l I \ r=t 



h(r)\ \&(t) 



K(r)+2/ fc (r)] |0(t) 



By Jensen's inequality, we have that e|t)| < B, where B 
is a constant that satisfies for all t: 

B > ~ J2 E { b k {tf } + \ J2 e { K (*) + y fe (*) ) 2 } 



fe=l 



fc=l 



Ee{»W0) 2 } 



Z=l 



Such a finite constant exists by the second moment bound- 
edness assumptions. The result of d40l > follows by adding the 
following term to both sides: 



t+T-1 



V E E{f(x(r))m)} 



T=t 



□ 
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